Image processing device, image processing method, and storage medium

ABSTRACT

An image processing device includes an input/output interface, and at least one processor. The at least one processor executes: detecting feature points in a facial image of an object included in an image taken from the input/output interface; acquiring first distance information between the feature points before a beauty treatment is performed; acquiring second distance information between the feature points at a second timing after the beauty treatment is performed; acquiring a difference value between the first distance information and the second distance information; and determining whether the beauty treatment is correctly given or has been given based on a different value between the acquired distance information.

TECHNICAL FIELD

The present invention relates to an image processing device, an imageprocessing method, and an image processing program.

BACKGROUND ART

For determination of whether a facial massage is correctly given duringa beauty treatment, it is required to measure the position of the faceand how hard the face is pressed with fingers or hands. In this case,pressure information and positional information need to be acquired fromsensors or markers attached to the fingers or the hands. This requires acomplicated system. Image recognition using a camera cannot clearlydistinguish the fingers and the hands from the color of the face, andthus, has difficulty in increasing the accuracy of pieces of informationincluding the positional information.

Patent Document. 1 discloses an appliance equipped with a mirror. Thisappliance can extract feature points defining the degree of fatigue of auser from a facial image using a feature extraction unit, and can showinformation about the degree of fatigue calculated from the featurepoints on the mirror.

-   Patent Document 1: Japanese Unexamined Patent Application,    Publication No. 2012-152389

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

According to the technology of Patent Document 1, a comparison is madebetween facial images before and after the beauty treatment, such as amassage. For this comparison, a display simply shows the facial imagesbefore and after the massage side by side, and the user just comparesthe two facial images to check the effect of the massage. That is, withthe technology disclosed by Patent Document 1, it is impossible to checkwhether the beauty treatment is correctly given during the beautytreatment, and quantitatively evaluate the effect of the beautytreatment.

An object of the present invention is to provide an image processingdevice, an image processing method, and an image processing program thatallow checking of whether a beauty treatment is correctly given duringthe beauty treatment, and/or quantitative evaluation of the effect ofthe beauty treatment.

Means for Solving the Problems

To achieve the object, an aspect of the present invention is directed toan image processing device, including: an acquisition means thatacquires an image taken of an object; a detection means that detects afeature point in a real image of the object included in the image of theobject; a comparison means that compares a locus obtained as a result oftracking of movement of the feature point in the real image included inthe acquired image of the object with preset locus information toacquire a comparison result; and an output means that outputs thecomparison result.

Effects of the Invention

The image processing device, image processing method, and imageprocessing program of the present invention allow checking of whether abeauty treatment is correctly given during the beauty treatment, and/orquantitative evaluation of the effect of the beauty treatment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an image processingsystem according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating an appearance of an image processingdevice according to an embodiment of the present invention as viewedfrom the front;

FIGS. 3A and 3B are diagrams illustrating an appearance of the imageprocessing device according to the embodiment of the present inventionas viewed from the side;

FIG. 4 is a block diagram illustrating a configuration of hardware ofthe image processing device according to the embodiment of the presentinvention;

FIG. 5 is a functional block diagram illustrating one, for executingmeasurement processing, of functional configurations of the imageprocessing device according to the embodiment of the present invention;

FIG. 6 is a diagram illustrating an example of feature points, referencepoints, and reference lines detected by the image processing deviceaccording to the embodiment of the present invention;

FIG. 7 is a diagram illustrating examples of reference points andreference lines used for different treatments according to theembodiment of the present invention.

FIG. 8A is a diagram illustrating how a correction is made according tothe embodiment of the present invention;

FIG. 8B is a diagram illustrating how a correction is made according tothe embodiment of the present invention;

FIGS. 9A and 9B are diagrams illustrating an example of a distancebetween a reference point and a reference line and preset distanceinformation according to the embodiment of the present invention;

FIGS. 10A and 10B are diagrams illustrating an example of the distancebetween the reference point and the reference line and the presetdistance information according to the embodiment of the presentinvention;

FIGS. 11A and 11B are diagrams illustrating an example of the distancebetween the reference point and the reference line and the presetdistance information according to the embodiment of the presentinvention;

FIGS. 12A and 12B are diagrams illustrating an example of the distancebetween the reference point and the reference line and the presetdistance information according to the embodiment of the presentinvention;

FIG. 13A is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention;

FIG. 13B is a flowchart of operation of the image processing deviceaccording to the embodiment of the present inventions FIG. 130 is aflowchart of operation of the image processing device according to theembodiment of the present invention;

FIG. 13D is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention;

FIG. 13E is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention;

FIG. 13F is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention;

FIG. 14A is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention; and

FIG. 14B is a flowchart of operation of the image processing deviceaccording to the embodiment of the present invention.

PREFERRED MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described below withreference to the drawings.

[Outline of Embodiment]

An image processing device 1 according to an embodiment of the presentinvention is a smart mirror constituted as a portable and freestandingmirror. The image processing device 1 takes an image of a user who is anobject seeing the mirror. Based on the image taken of the user,particularly an image taken of a face of the user, the image processingdevice 1 detects feature points from the user's face, and continuouslytakes the image of the user's face to track the movement of the featurepoints. The image processing device 1 compares a locus obtained as aresult of the tracking with preset locus information to output whether abeauty treatment is correctly performed by the user during the beautytreatment, and the effect of the treatment.

[System Configuration]

FIG. 1 is a block diagram illustrating an overall configuration of animage processing system I including the image processing device 1 of thepresent embodiment. As shown in FIG. 1, the image processing system Iincludes a plurality of image processing devices 1, a network 2, and aserver group 3. The number of image processing devices 1 is not limitedto particular values. The image processing system I may include n (n isany natural number) image processing devices 1. In the followingdescription, the n image processing devices 1 will be simply, referredto as the “image processing device 1” without adding a letter of thealphabet to the reference numeral when no distinction between the nimage processing devices 1 is required.

The image processing device 1 detects the feature points of the userfrom an image, tracks the movement of the feature points detected,compares the locus obtained as a result of the tracking with the presetlocus information, and displays the comparison result. The imageprocessing device 1 is connected to servers included in the server group3 via the network 2 to be able to mutually communicate with the servers.

The network 2 is achieved, for example, by any one of the Internet, alocal area network (LAN), or a mobile telephone network, or acombination of them.

The server group 3 includes various servers that cooperate with theimage processing device 1. For example, the server group 3 includes anauthentication server that authenticates the user of the imageprocessing device 1. For example, the server group 3 also includes anapplication distribution server that achieves one of functions of theimage processing device 1. For example, the server group 3 furtherincludes a measurement data storage server that stores user profileinformation, which is information including setting information relatedto the user, and a history of usage of the image processing device 1 bythe user.

The image processing system I shown in FIG. 1 is merely an example. Theserver group 3 may include a server having a different function. Theservers included in the server group 3 may be formed as independentserver devices, or a single server device.

[Configuration of Appearance]

FIG. 2 is a diagram illustrating an appearance of the image processingdevice 1 according to the embodiment of the present invention as viewedfrom the front. FIG. 3 is a diagram illustrating an appearance of theimage processing device 1 as viewed from the side. A front surface ofthe image processing device 1 has a size of A4 defined by aninternational standard, e.g., International Organization forStandardization (ISO) 216.

As shown in FIGS. 2 and 3, the image processing device 1 includes a body30, a leg 31, and hinges 32. The body 30 is a portion including adisplay 18 and other pieces of hardware which will be described laterwith reference to FIG. 4. The leg 31 and the hinges 32 are members thatallow the image processing device 1 to be freestanding. The leg 31 issupported by the hinges 32 to be rotatable with respect to the body 30.

As shown in FIG. 3A, when carrying the image processing device 1, theuser can align the side surface of the body 30 with the side surface ofthe leg 31 to reduce the size of the image processing device 1. Whenusing the image processing device 1 on a desk, the user rotates the leg31 about the hinges 32 so that the image processing device 1 can befreestanding as shown in FIG. 3B. The hinges 32 have a mechanism forholding the leg 31 at a predetermined angle so that the image processingdevice 1 can be freestanding.

The body 30 includes the display 18 as described above. The display 18is a portion on which various types of information are shown to theuser. The display 18 shows, for example, a user image which is a realimage of the user, who is a subject, taken by an imaging unit 16(corresponding to a user image 51 in the drawing), an avatar image whichis an alternative image of the user (corresponding to an avatar image 52in the drawing), a guiding image which is auxiliary information forgiving guidance (corresponding to a guiding image 53 in the drawing),and a biometric information image indicating biometric information ofthe user (corresponding to a biometric information image 54 in thedrawing). In this case, the display 18 shows t-e guiding image combinedwith, and superposed on, the avatar image.

The user seeing the display 18 can recognize various types ofinformation at a time. As described above, those shown on the display 18have a sense of unity suitable for the user to see, with no discomfortin how they look.

As shown in FIG. 2, the image processing device 1 further includes animaging unit 16, an input unit 17, and the display 18 as componentsforming the appearance.

The imaging unit 16 is a portion that takes an image of a subject, i.e.,the user who faces the display 18 when using the image processing device1. The imaging unit 16 is positioned to be able to take an image of theface of the user in front of the display 18. For example, the imagingunit 16 is arranged on the front surface of the body 30 above thedisplay 18 as shown in the drawing.

The input unit 17 is a portion that accepts input from the user. Theinput unit 17 includes, for example, a plurality of buttons. The drawingshows, as examples, switching buttons used for switching among variousmodes, such as a face-slimming massage, smile training, and recording ofbiometric information, and a button for turning the power of the imageprocessing device 1 on/off.

The components forming the appearance of the image processing device 1are described above. Note that the components are merely examples, andare not limited to these examples.

For example, the image processing device 1 may further include a lightemitter that illuminates the user facing the display 18. When the lightemitter throws light having adjusted illuminance and color components onthe user, the image processing device 1 functions as an illuminatedmirror. The image processing device 1 may have two or more lightemitters. The light emitter may be arranged above or below the display18, or may surround the display 18.

The number and position of the input unit 17 may be changed. Forexample, part of the display 18 may be formed as a touch screen so thatthe input unit 17 and the display 18 are incorporated into a singleunit.

[Hardware Configuration]

FIG. 4 is a block diagram illustrating a configuration of hardware ofthe image processing device 1. As shown in FIG. 4, the image processingdevice 1 includes a central processing unit (CPU) 11 which is aprocessor, a read only memory (ROM) 12, a random access memory (RAM) 13,a bus 14, an input/output interface 15, the imaging unit 16, the inputunit 17, the display 18, a storage 19, a communication unit 20, a drive21, and a battery 22.

The CPU 11 executes various types of processing in accordance withprograms stored in the ROM 12 or loaded from the storage 19 to the RAM13.

The RAM 13 also stores data required for the CPU 11 to execute varioustypes of processing as appropriate.

The bus 14 mutually connects the CPU 11, the ROM 12, and the RAM 13. Theinput/output, interface 15 is also connected to the bus 14. Theinput/output interface 15 is connected to the imaging unit 16, the inputunit 17, the display 18, the storage 19, the communication unit 20, thedrive 21, and the battery 22.

Although not shown, the imaging unit 16 includes an optical lens and animage sensor. The optical lens is comprised of a lens that gatherslight, such as a focus lens and a zoom lens, for taking an image of asubject. The focus lens forms an image of the subject on aphotosensitive surface of the image sensor. The zoom lens freely changesa focal length in a certain range. The imaging unit 16 further includesa peripheral circuit that adjusts setting parameters, such as a focalpoint, exposure, and white balance, as needed.

The image sensor is comprised of, for example, a photoelectricconversion element and an analog front end (APE). The photoelectricconversion element includes, for example, a complementary metal oxidesemiconductor (CMOS). An image of the subject enters the photoelectricconversion element from the optical lens. The photoelectric conversionelement photoelectrically converts the image of the subject (takes theimage), accumulates an image signal for a certain period of time, andthen successively supplies the accumulated image signal as an analogsignal to the AFE. The AFE executes various types of signal processing,such as analog/digital (A/D) conversion processing, on the analogsignal. A digital signal is generated by the various types of signalprocessing, and outputted as an output signal of the imaging unit 16.The output signal of the imaging unit 16 is suitably supplied to thecomponents, such as the CPU 11.

The input unit 17 includes various buttons and a microphone, and entersvarious types of information in response to an instruction by the user'soperation or voice.

The display 18 is comprised of a liquid crystal display, and shows animage corresponding to the image data outputted by the CPU 11.

The storage 19 is comprised of a semiconductor memory, such as dynamicrandom access memory (DRAM), and stores various types of data.

The communication unit 20 performs communication control so that the CPU11 communicates with other devices (e.g., the servers included in theserver group 3) via the network 2.

The drive 21 is comprised of an interface that can accept a removablemedia 100. The drive 21 suitably accepts the removable media 100, suchas a magnetic disc, an optical disc, a magneto-optical disc, or asemiconductor memory. The removable media 100 stores programs forexecuting composite display processing which will be described later,and various data including the image data. The programs and various datasuch as the image data read from the removable media 100 by the drive 21are stored in the storage 19 as needed.

The battery 22 supplies power to the components, and is chargeable whenconnected to an external power supply. When the image processing device1 is not connected to the external power supply, the power from thebattery 22 operates the image processing device 1.

The image processing device 1 may further include other pieces ofhardware in addition to the above-described ones. For example, the imageprocessing device 1 may include a lamp, a speaker, or a vibration motor,and may further include an output unit that outputs light, sound, or avibration signal.

[Functional Configuration]

FIG. 5 is a functional block diagram illustrating one, for executingimage processing, of functional configurations of the image processingdevice 1. Image processing is a series of processing executed to displayan image based on a change of feature points acquired from the user bythe image processing device 1.

The storage 19 that stores various types of information will bedescribed first. As shown in FIG. 5, an area of the storage 19 includesa guidance information storage 191, an avatar information storage 192,an actual measurement information storage 193, and an evaluation resultinformation storage 1944.

The guidance information storage 191 stores various types of datarelated to guidance in the display processing. For example, the guidanceinformation storage 191 stores guiding image data, text data, and sounddata for generating guidance information to be shown in the displayprocessing. In particular, the guidance information storage 191 of thepresent embodiment stores locus information, which is information abouta locus that the feature points are supposed to form during a facialmassage given as a beauty treatment. The guidance information storage191 also stores distance information, which is information about anideal distance between the feature points. More specifically, thedistance information may be information about an ideal distance thatvaries depending on partial change in shape of a face caused by themassage. The guidance information storage 191 also stores timinginformation about desirable timing when the movement of the massagechanges. The guidance information storage 191 also stores the order ofdisplay of many items of guidance information for presenting a series ofguidance, a condition for switching the displayed guidance informationto next guidance information, and data for generating various userinterfaces. The guidance information storage 191 also stores userprofile information generated in connection with the display processing.The guiding image may be image data previously generated, or may begenerated by computer graphics (CG) in real time based on calculation.

The avatar information storage 192 stores various types of data relatedto an avatar which is an alternative of a real image of the user. Forexample, the avatar information storage 192 stores avatar image data forgenerating an avatar image shown in composite display processing, and acondition for selecting an avatar image to be shown in the guidance. Theavatar image may be image data previously generated, or may be generatedby CC; in real time based on calculation. The avatar image may be, forexample, an image of a character that imitates a human or an animal. Anynumber of avatar images may be prepared, and a suitable one may beselected depending on the characteristics of the user (e.g., sex, age,and preference of the user) or what the guidance tells. The avatar imagemay be continuous images for creating animation of a character. Eachdata may be stored as a single library per type of guidance so that thedata is easily read out for creating a series of guidance.

The actual measurement information storage 193 stores pieces of actualmeasurement information about the feature points detected by a detectionunit 113, positions of reference points included in the feature points,and a distance between the feature points calculated by a comparisonunit 115. In particular, the actual measurement information storage 193may store, as the distance information, initial values of coordinates ofthe feature points including the reference points, and an initial valueof the distance between the feature points before the massage as thebeauty treatment starts. More specifically, the distance information maybe information about a distance that varies depending on partial changein shape of a face caused by the massage.

The evaluation result information storage 194 stores information that anevaluation unit 116 uses to generate evaluation result information basedon a result of comparison between the result of tracking of the movementof the feature points and the preset locus information by the comparisonunit 115, information for displaying the evaluation result, andinformation indicating the evaluation result. Evaluation processing theevaluation unit 116 will be described later.

The pieces of information stored in the guidance information storage191, the avatar information storage 192, the actual measurementinformation storage 193, and the evaluation result information storage194 may be stored in the storage 19 only, or may be suitably stored inthe removable media 100 by the drive 21. The pieces of informationstored in the guidance information storage 191, the avatar informationstorage 192, the actual measurement information storage 193, and theevaluation result information storage 194 may be suitably stored in ameasurement data storage server included in the server group 3.

Functional blocks for executing the image processing will be describedbelow. As shown in FIG. 5, a setting processing unit 111, an acquisitionunit. 112, a detection unit 113, a correction unit 114, a comparisonunit. 115, an evaluation unit. 116, and a display control unit 117function in the CPU 11.

The setting processing unit 111 controls settings related to the imageprocessing and the display processing. The setting processing unit 111acquires application software for executing the display processing from,for example, the application distribution server included in the servergroup 3, and runs the application software. The setting processing unit111 communicates with, for example, the authentication server includedin the server group 3, to authenticate the user who performs the displayprocessing. The setting processing unit 111 communicates with, forexample, the measurement data storage server included in the servergroup 3, to update the user profile information in the displayprocessing.

The setting processing unit 111 displays a menu for presenting theguidance based on the application software for executing the displayprocessing. For example, the setting processing unit 111 displays a menuincluding options to choose the contents of the guidance, such as“face-slimming massage”, “smile training”, “measurement of biometricinformation”, and “makeup”. The setting processing unit 111 receives anentry of the contents of guidance chosen by the user who referred to themenu via the input unit 17. For example, the setting processing unit 111receives that the “face-slimming massage” is chosen. Then, the displayprocessing is executed to provide the guidance about the face-slimmingmassage. The face-slimming massage is a massage given to the user's faceby the user, such as a lymphatic massage that reduces the swelling ofthe face by lymphatic drainage.

The acquisition unit 112 acquires an image of an object taken by theimaging unit 16. In the present embodiment, the “object” is preferably,but not limited to, an organ of a human, particularly a face of a human.

The detection unit 113 detects the feature points in a real imageincluded in the image of the object acquired by the acquisition unit112. In the present embodiment, the “image of the object” is an imageincluding the object itself and its background, and indicates an imagingrange (imaging angle of view) of the imaging unit 16. The “real image”is the object itself (or an area of the image indicating the object).Especially in the present embodiment, the “image of the object” issuitably an image including an image of a human's face, and the “realimage of the object” is suitably a real image of the human's face, butthese images are not limited to such examples.

The “feature points” are points (or areas) provided to detect andrecognize the shape of organs (eyebrows, eyes, nose, mouth, facialcontour) in the image of the user's face.

FIG. 6 shows an example of the feature points, in particular, thereference points and reference lines which are used as a reference forthe detection of the movement of the feature points. As shown in FIG. 6,the feature points are arranged along the contour of the eyes, along thedorsum and wings of the nose, around the mouth, and along the eyebrowsand the contour of the cheeks extending from the eyebrows. Among thefeature points, two at the inner corners of the eyes (E1 and E2 in FIG.6) and two at the corners of the mouth (M1 and M2 in FIG. 6) arepossible reference points. A straight line connecting E1 and E2(reference line A) and a straight line connecting N1 and N2 (referenceline B) are possible reference lines. A distance between E1 and E2 isthe distance information described above. When the beauty treatment,e.g., a massage, changes the distance between E1 and E2, the changeddistance is compared with the distance information initially set.

FIG. 7 shows examples of the reference points and the reference linesused for different treatments. For a massage around the eyes, thefeature points corresponding to two outer corners of the eyes enclosedin circles in FIG. 7 (E3 and E4) will be the reference points.

For a massage of the temple, the feature points corresponding to twoends of the eyebrows enclosed in circles in FIG. 7 (T1 and T2) will bethe reference points.

For a massage around the nose, the two feature points corresponding tothe ends of the nasal wings of the nose enclosed in circles in FIG. 7(N3 and N4) will be the reference points.

For a massage of the cheeks and the periphery of the mouth, the twofeature points corresponding to the corners of the mouth enclosed incircles in FIG. 7 (M1 and M2) will be the reference points.

For a massage of the whole face, the uppermost two of the feature pointsalong the contour of the cheeks enclosed in circles in A. 7 (F1 and F2)and the lowermost two of the feature points along the contour of thecheeks enclosed in circles in FIG. 7 (F3 and F4) will be the referencepoints.

The correction unit 114 performs scaling of the taken real image, forexample, when the user's face (real image) taken by the imaging unit 16is inclined at a predetermined angle of rotation, or depending on thedistance between the imaging unit 16 and the user's face. Morespecifically, the correction unit 114 corrects the positionalrelationship among the plurality of feature points detected by thedetection unit 113. FIGS. 8A and 8B show how the correction is made bythe correction unit 114. For correction of the tilt of a horizontalline, an angle formed by the reference line connecting E1 and E2 and ahorizontal straight line is detected as shown in FIG. 8A, and the imageis rotated to reduce the angle to zero. For correction of the tilt of avertical line, an angle formed by the reference line connecting N1 andN2 and a vertical straight line is detected as shown in FIG. 8B, and theimage is rotated to reduce the angle to zero. For correction of the sizeof the image, the image is magnified or reduced so that the distancebetween E1 and E2 becomes a preset reference distance.

The comparison unit 115 tracks the movement of the feature pointsdetected by the imaging unit 16 continuously taking the image of theobject, and compares a locus obtained as a result of t e tracking withpreset locus information. The comparison unit 115 may compare the timingof the movement with preset timing information.

e comparison unit 115 may obtain a distance between the feature pointsdetected, and compare the distance with the preset distance information.For example, the comparison unit 115 may compare the distance betweenthe reference point and the reference line with the preset distanceinformation.

FIGS. 9A to 12B show examples of the distance between the referencepoint and the reference line and the preset distance information, whichare compared by the comparison unit 115.

FIG. 9A shows an example of the movement of fingers giving a massage. Inthis example, the massage is given to the cheeks and the periphery ofthe mouth as described with reference to FIG. 7. In this massage,specifically, the user first pushes the cheeks up in the direction (1)with the fingers placed near the mouth, moves the right fingers to theright and the left fingers to the left so that the cheeks spread in thedirections (2), and finally moves the fingers in the directions (3) sothat the fingers moved to the right and the left return to the originalpositions while pushing the cheeks toward the center of the face.

Points M1 and M2 in the real image taken by the imaging unit. 16 beforegiving the massage are set as the reference points as shown in FIG. 9B.A distance from M1 to the reference line connecting the two outercorners of the eyes is referred to as VR0, and a distance from M2 to thesame reference line as VL0. A distance from M1 to the reference linepassing the dorsum of the nose is referred to as HR0, and a distancefrom M2 to the same reference line as HL0. The comparison unit 115stores the pieces of distance information thus defined in the actualmeasurement information storage 193.

When the massage as the treatment starts, the fingers move in thedirection (1) as shown in FIG. 10A. Then, the distances change as shownin FIG. 10B, i.e., VR0 changes to VR1, VL0 to VL1, HR0 to HR1, and HL0to HL1. The changed distances are compared to the pieces of distanceinformation. In the example shown in FIG. 10B, VR0>VR1, VL0>VL1,HR0≤HR1, and HL0≤HL1 are met.

Then, when the fingers move in the directions (2) as shown in FIG. 11A,VR1 changes to VR2, VL1 to VL2, HR1 to HR2, and HL1 to HL2 as shown inFIG. 11B. The changed distances are compared to the pieces of distanceinformation. In the example shown in FIG. 11B, VR0≤VR2, VL0≤VL2,HR0<HR2, and HL0<HL2 are met.

Finally, when the fingers move in the directions (3) as shown in FIG.12A, VR2 changes to VR3, VL2 to VL3, HR2 to HR3, and HL2 to as shown inFIG. 12B. The changed distances are compared to the pieces of distanceinformation. In the example shown in FIG. 12B, VR0=VR3, VL0=VL3,HR0=HR3, and HL0≥HL3 are met. Regarding the movement of the featurepoints, the determination of “=”, “≤”, and “≥” does not indicate thatthe positions of the feature points are strictly the same, and takes acertain degree of errors into consideration.

The evaluation unit 116 evaluates the results of comparison by thecomparison unit 115, and generates an evaluation result. For example,the evaluation unit 116 may generate the evaluation result based on thedegree of coincidence between the locus obtained as a result of thetracking of the movement of the feature points and the preset locusinformation. The evaluation unit 116 may generate the evaluation resultbased on the degree of coincidence between the timing of the movement ofthe feature points and the preset timing information. The evaluationunit 116 may generate the evaluation result based on the degree ofcoincidence between the distance between the plurality of feature pointsdetected and the distance information. The evaluation unit 116 may storethese evaluation results in the evaluation result information storage194, and may generate another evaluation result based on the history ofthe evaluation results stored in the evaluation result informationstorage 194.

The display control unit 117 serving as an output means executesprocessing to output the comparison results, the evaluation results, andthe history of the evaluation results to the display 18. In thefollowing description, the display control unit 117 may be sometimesreferred to as an “output unit 117”.

The display control unit 117 also executes processing for switchingbetween a first display mode and a second display mode. In the firstdisplay mode, an image of the user is displayed as a primary image,together with a secondary image which is a composite image of an avatarimage and a guiding image. In the second display mode, the image of theuser is displayed as a secondary image, together with the compositeimage of the avatar image and the guiding image as the primary image. Asshown in FIG. 2, the composite image of the avatar image and the guidingimage can be shown in a large size in the center of the screen as theprimary image, together with the image of the user shown in a smallersize than the primary image in a lower portion of the screen as thesecondary image. Conversely, the image of the user can be shown in alarge size in the center of the screen as the primary image, togetherwith the composite image of the avatar image and the guiding image shownin a smaller size than the primary image in a lower portion of thescreen as the secondary image.

The display control unit 117 may automatically show, as an image ortext, a direction to which the user is supposed to turn the face, how tomassage, and the biometric information in a region not overlapping witha mirror image of the user's face or the face of the avatar image duringa series of guidance. The display control unit 117 may output theguidance information by a different method together with the displayedinformation. For example, the display control unit 117 may read guidanceinformation including sound data, and may output sound or musicgenerated from the read guidance information. The display control unit.117 may change, for example, the state of lighting by the light emitter.

[Image Processing]

FIGS. 13A to 13F are flowcharts illustrating an example of an overallflow of first image processing executed by the image processing device 1of FIG. 1 having the functional configuration shown in FIG. 5.

In Step S1, the setting processing unit 111 performs initial setting.For example, the setting processing unit 111 downloads personal datawhich is set for each user ID from the authentication server included inthe server group 3.

In Step S2, the setting processing unit 111 executes start processing todisplay a menu screen on the display 18.

If the user chooses the massage and the choice is detected in Step S3(the answer is YES in S3), the process proceeds to Step S4. If the userdoes not choose the massage yet and no choice is detected (the answer isNO in S3), the process repeats Step S3.

In Step S4, the display control unit 117 allows the display 18 to show a“face position guide” with which the user aligns the face to do themassage.

In Step S5, the detection unit 113 detects the coordinates of thefeature points on the face in a normal state (including the referencepoints E1, E2, N1, N2, M1, and M2 in the example described above).

In Step S6, the comparison unit 115 calculates a distance between eachof the reference points and an associated one of the reference lines(Ro, VL0, HR0, HL0, and HE0 in the example described above).

In Step S7, if the detection of the feature points is completed (theanswer is YES in S7), the process proceeds to Step S8. If the detectionof the feature points is not completed (the answer is NO in Step S7),the process proceeds to Step S5.

In Step S8, the comparison unit 115 stores the coordinates of thereference points detected in Step S5 and data of the distancesassociated with the reference points calculated in Step S6 in the actualmeasurement information storage 193.

In Step S9, the setting processing unit 111 sets a score value S, whichis the evaluation result and counted by a score counter, to zero.

In Step S10, the setting processing unit 111 sets a repeat count valueL, which is a repeat count of the massage and counted by a repeatcounter, to zero.

In Step S11, the display control unit 117 displays guidance of themassage with a CG avatar.

In Step S12, the display control unit 117 displays an instruction tostart the massage to the user.

In Step S13, the setting processing unit 111 sets timing values (C) inthe whole massage. Specifically, the timing value after the massage ofFIG. 10A is set to C=1, the timing value after the massage of FIG. 11Ato C=2, and the timing value after the massage of FIG. 12A to C=3. Atpresent, the timing value counted by the timing counter is set to zerobecause the massage is not started yet.

In Step S14, the display control unit 117 displays the timing of themassage on the display 18, and provides audio guidance from a speakingunit (not shown).

In Step S15, if the timing value C counted by the timing counter is anyone of zero to two (the answer is zero to two in S15), the processproceeds to Step S16. If the timing value C is three (the answer isthree in S15), the process proceeds to Step S17.

In Step S16, the detection unit 113 detects the coordinates of thefeature points (reference points E1, E2, N1, N2, M1, and M2 in theexample described above).

In Step S17, one is added to the repeat count value L counted by therepeat counter. Thereafter, the process proceeds to Step S13.

In Step S18, the correction unit 114 corrects the size of a set of thefeature points using the reference points on the face in the normalstate. In this case, the correction is made so that the distance HE(=E2−E1) becomes HE0 in the normal state.

In Step S19, the correction unit 114 corrects the tilt of the set of thefeature points using the reference points on the face in the normalstate. More specifically, the correction unit. 114 makes a correction sothat the horizontal line passing E1 and E2 is level and the verticalline passing N1 and N2 is plumb.

In Step S20, the comparison unit 115 calculates a distance related toeach of the reference points (VR1, VL1, HR1, and HL1 in the exampledescribed above).

In Step S21, if the detection by the detection unit 113 is completed(the answer is YES in S21), the process proceeds to Step S22. If thedetection is not completed (the answer is NO in S21), the processproceeds to Step S16.

In Step S22, the value C is detected. If C=0, (the answer is zero inS22), the process proceeds to Step S23. If the value is other than zero,e.g., C is any one of one or two (the answer is one or two in S22), theprocess proceeds to Step S27.

In Step S23, the treatment in the direction (1) shown in FIG. 9A isperformed. If the positions of the feature points VR0, VL0, HR0, and HL0are shifted to meet VR0>VR1, VL0>VL1, HR0≤HR1, and HL0≤HL1 (the answeris YES in S23), the process proceeds to Step S24. Otherwise (the answeris NO in S23), the process proceeds to Step S25.

In Step S24, the comparison unit 115 calculates the rate of change VRfrom VR0 to VR1 and the rate of change VL from VL0 to VL1 from theformulae (1) and (2).

VR=(1−(VR1/VR0))*100  (1)

VL=(1−(VL1/VL0))*100  (2)

Thereafter, the process proceeds to Step S36.

In Step S25, if the treatment in the direction (1) shown in FIG. 9A goeson beyond a predetermined time limit (the answer is YES in S25), theprocess proceeds to Step S26. If the treatment ends before the timelimit (the answer is NO in S25), the process proceeds to Step 23.

In Step S26, the evaluation unit 116 subtracts one from the score valueS which is the evaluation result. Thereafter, the process proceeds toStep S14.

In Step S27, if the value C is one, (the answer is one in S27), theprocess proceeds to Step S28. If the value C is two, (the answer is twoin S27), the process proceeds to Step S32.

In Step S28, the treatment in the directions (2) shown in FIG.performed. If the positions of the feature points VR0, VL0, HR0, and HL0are shifted to meet VR0≤VR2, VL0≤VL2, HR0<HR2, and HL0<HL2 (the answeris YES in S28), the process proceeds to Step S29. Otherwise (the answeris NO in S28), the process proceeds to Step S30.

In Step S29, the comparison unit 115 calculates the rate of change HRfrom HR0 to HR2 and the rate of change HL from HL0 to HL2 from theformulae (3) and (4).

HR=1−(HR2/HR0))*100  (3)

HL=(1−(HL2/HL0))*100  (4)

Thereafter, the process proceeds to Step S36.

In Step S30, if the treatment in the directions (2) shown in FIG. 9Agoes on beyond a predetermined time limit (the answer is YES in S30),the process proceeds to Step S31. If the treatment ends before the timelimit (the answer is NO in S30), the process proceeds to Step S28.

In Step S31, the evaluation unit 116 subtracts one from the score valueS which is the evaluation result. Thereafter, the process proceeds toStep S14.

In Step S32, the treatment in the directions (3) shown in FIG. 9A isperformed. If the positions of the feature points VR0, VL0, HR0, and HL0are shifted to meet VR0=VR3, VL0=VL3, HR0≥HR3, and HL0≥HL3 (the answeris YES in S32), the process proceeds to Step S33. Otherwise (the answeris NO in S32), the process proceeds to Step S34.

In Step S33, the comparison unit 115 calculates the rate of change HRfrom HR3 to HR0 and the rate of change HL from HL3 to HL0 from theformulae (5) and (6).

HR=(1−(HR0/HR3))*100  (3)

HL=(1−(HL0/HL3))*100  (4)

Thereafter, the process proceeds to Step S36.

In Step S34, if the treatment in the directions (3) shown in FIG. 9Agoes on beyond a predetermined time limit (the answer is YES in S34),the process proceeds to Step S35. If the treatment ends before the timelimit (the answer is NO in S34), the process proceeds to Step S32.

In Step S35, the evaluation unit 116 subtracts one from the score valueS which is the evaluation result. Thereafter, the process proceeds toStep S14.

If VR>10% and VL>10% are met in Step S36 (the answer is YES in S36), theprocess proceeds to Step S37. Otherwise (the answer is NO in S36), theprocess proceeds to Step S38.

In Step S37, the evaluation unit 116 adds 10 to the score value S whichis the evaluation result. Thereafter, the process proceeds to Step S41.

If VR>5% and VL>5% are met in Step S38 (the answer is YES in S38), theprocess proceeds to Step S39. Otherwise (the answer is NO in S38), theprocess proceeds to Step S40.

In Step S39, the evaluation unit 116 adds five to the score value Swhich is the evaluation result. Thereafter, the process proceeds to StepS41.

In Step S40, the evaluation unit 116 adds one to the score value S whichis the evaluation result. Thereafter, the process proceeds to Step S41.

In Step S41, the display control unit 117 displays the score value S,and the speaking unit (not shown) speaks the score value S.

In Step S42, the setting processing unit 111 adds one to the timingvalue C counted by the timing counter.

In Step S43, the comparison unit 115 stores the coordinates of thereference points and data of the measurements of the distancesassociated with the reference points in the actual measurementinformation storage 193.

If the repeat count value L counted by the repeat counter exceeds threein Step S44 (the answer is YES in S44), the process proceeds to StepS45. If the repeat count value L does not exceed three (the answer is NOin S44), the process proceeds to Step S14.

If the massage ends in Step S45 (the answer is YES in S45), the processproceeds to Step S46. Otherwise (the answer is NO in S45), the processproceeds to Step S3.

In Step S46, the image processing device 1 turns the power OFF.

FIGS. 14A and 14B are flowcharts illustrating an example of an overallflow of second image processing executed by the image processing device1 of FIG. 1 having the functional configuration shown in FIG. 5. Thesecond image processing is executed when the user does the massage for along period of time.

In Step S51, the setting processing unit 111 queries the server group 3to perform personal authentication of the user.

If the user is authenticated in Step S52 (the answer is YES in S52), theprocess proceeds to Step S53. If the user is not authenticated (theanswer is NO in S52), the process proceeds to Step S51.

In Step S53, the setting processing unit 111 reads the target number ofdays the user tries to do the massage in succession from the servergroup 3.

In Step S54, the setting processing unit 111 reads the number ofconsecutive days the user did the massage in succession from the servergroup 3 based on a done flag which is a flag added to the history of theactual measurement data and indicates whether the massage is done.

If the number of consecutive days is less than the target number of daysin Step S55 (the answer is YES in S55), the process proceeds to StepS56. If the number of consecutive days reaches or exceeds the targetnumber of days (the answer is NO in S55), the process proceeds to StepS57.

In Step S56, the display control unit 117 shows on the display 18 thenumber of days left until the target date, i.e., the number of daysobtained by subtracting the number of consecutive days from the targetnumber of days.

In Step S57, the setting processing unit 111 adds the target number ofdays. For example, the setting processing unit 111 adds 28 days, i.e.,four weeks, to the target number of days at present.

If the start of the treatment is detected in Step S58 (t answer is YESin S58), process proceeds to Step S59. If the start of the treatment isnot detected (the answer is NO in S58), the process proceeds to StepS58.

In Step S59, the image processing device 1 executes the processing ofSteps S1 to S46 shown in FIGS. 13A to 13F as beauty treatmentdetermination processing.

If the end of the treatment is detected in Step S60 (the answer is YESin S60), the process proceeds to Step S61. If the end of the treatmentis not detected (the answer is NO in S60), the process proceeds to StepS59.

In Step S61, the evaluation unit 116 stores the score vaiue after theend of the treatment in the evaluation result information storage 194.

If this score value S is at a certain level or above in Step S62 (theanswer is YES in S62), the process proceeds to Step S63. If this scorevalue S falls below the certain level (the answer is NO in S62), theprocess proceeds to Step S64.

In Step S63, the setting processing unit 111 stores a flag indicatingthat the treatment that has brought the score value S to a certain levelor above is done in the server group 3.

In Step S64, the setting processing unit 111 calculates the number ofconsecutive days the treatment was done.

If the number of consecutive days reaches or exceeds the target numberof days in Step S65 (the answer is YES in S65), the process proceeds toStep S66. If the number of consecutive days is less than the targetnumber of days (the answer is NO in S65), the process proceeds to StepS67.

In Step S66, the display control unit 117 displays on the display 18that the user reached the target.

If the end of the treatment is detected in Step S67 (the answer is YESin S67), the process proceeds to Step S68. If the end of the treatmentis not detected (the answer is N1 in S67), the process proceeds to StepS58.

In Step S68, the setting processing unit 111 stores the target number ofdays in the server group 3.

The image processing device 1 of the present embodiment described aboveincludes the acquisition unit 112, the detection unit 113, thecomparison unit 115, and the output unit. (display control unit) 117.The acquisition unit 112 acquires the image taken of the object. Thedetection unit 113 detects the feature points in a real image of theobject included in the image taken of the target. The comparison unit.115 compares the locus obtained as a result of the tracking of themovement of the feature points in the real image included in the imageof the object acquired by the acquisition unit 112 with the preset locusinformation to acquire the comparison result. The output unit 117outputs the comparison result.

The comparison between the locus of the movement of the feature pointsof the object detected using a camera and the preset locus informationallows determination of whether the treatment is correctly given to theobject.

The detection unit 113 detects a plurality of feature points. Thecomparison unit 115 obtains a distance between the plurality of featurepoints detected, and compares the distance with the preset distanceinformation. Thus, how much the distance between the feature points haschanged from the initial set value can be acquired, and whether thetreatment is correctly given to the object can be determined based onthe acquired result.

The image is taken while the treatment is given to the object. Thedistance information is information about the distance that variesdepending on partial change in shape of the object caused by thetreatment. Thus, whether the treatment is correctly given can bedetermined based on how the shape of the object is changed by thetreatment.

The image processing device 1 of the present embodiment further includesthe evaluation unit 116. The evaluation unit 116 evaluates thecomparison result obtained by the comparison unit. 115, and generates anevaluation result. The output unit 117 further outputs the evaluationresult. Thus, how correctly the treatment is given can be expressed as ascore based on the change of the feature points.

The evaluation unit 116 evaluates the degree of coincidence between thedistance and the distance information. Thus, now correctly the treatmentis given can be expressed as a score based on how much the distancebetween the feature points coincides with the preset distanceinformation.

The image processing device 1 of the present embodiment further includesthe evaluation result information storage 194. The evaluation resultinformation storage 194 stores the history of the evaluation result. Theoutput unit 117 outputs the history of the evaluation result stored inthe evaluation result information storage 194. This allows the user ofthe image processing device 1 to thoroughly know the history of theevaluation result, i.e., the history of the score representing howcorrectly the treatment has been given so far.

When the number of consecutive days the treatment is evaluated to be atthe certain level or above by the evaluation unit 116 reaches or exceedsthe preset target number of days, the output unit 117 outputs that thenumber of consecutive days has reached or exceeded the target number ofdays. This motivates the user of the image processing device 1 tocontinue the treatment that is evaluated to be at the certain level orabove.

The comparison unit 115 compares the timing of the movement with thepreset timing information. This allows the user of the image processingdevice 1 to thoroughly know whether the treatment is done at a suitabletiming.

The object is an organ of a human. Thus, whether the treatment such as amassage is correctly given to the organ of the human can be determined.

Alternatively, the object is a face of a human. Thus, whether thetreatment such as a massage is correctly given to the face of the humancan be determined.

[Variations]

The present invention is not limited to the embodiment described above.Modifications and improvements within the scope that the object of thepresent invention can be achieved are included in the present invention.For example, the above-described embodiment can be modified as follows.

It has been described in the embodiment that the rate of change of thedistance between the feature points is compared with a predeterminedvalue, and how correctly the treatment is given is expressed as a scorebased on the comparison result. However, the comparison is not limitedto this example. For example, the length of the distance between thefeature points changed may be compared with a predetermined value, andhow correctly the treatment is given may be expressed as a score basedon the comparison result.

The image processing device 1 of the embodiment may have the display 18combined with a mirror having a reflective surface. In this case, themirror is comprised of a half mirror having optical properties,including both of transparency and reflectivity. The mirror issuperposed on the front side of the display 18 in the direction of theuser's sight. This arrangement allows the user to see, for example, theface of the user reflected in the mirror instead of the image of theuser taken by the imaging unit 16, together with various types ofinformation (e.g., a composite image) that are shown on the display 18and penetrate the mirror. Specifically, in the above-describedembodiment, the user sees the image of the user, who is the subject,taken by the imaging unit 16 as a real image of the user. However, inthis variation, the user sees a mirror image of the user reflected inthe mirror as the real image of the user. This configuration can providethe same advantages as the embodiment.

Other Modified Examples

For example, in the above embodiment, it is assumed that the imageprocessing device 1 cooperates with the respective servers included inthe server group 3, but the functions of the respective servers may beadded to the image processing device 1, and all the processes may beperformed only in the image processing device 1.

In addition, in the above embodiment, the image processing device 1 towhich the present invention is applied has been described by way ofexample of an electronic device incorporated in a portable self-standingmirror, but the present invention is not particularly limited thereto.For example, the present invention can be applied to an electronicdevice incorporated into a large mirror such as a full-length mirror, anelectronic device incorporated into a stationary bathroom vanity, and amirror-shaped electronic device installed in a bathroom.

The processing sequence described above can be executed by hardware, andcan also be executed by software. In other words, the functionalconfiguration of FIG. 5 is merely an illustrative example, and thepresent invention is not particularly limited thereto. Morespecifically, the types of functional blocks employed to realize theabove-described functions are not particularly limited to the examplesshown in FIG. 7, so long as the image processing device 1 can beprovided with the functions enabling the aforementioned processingsequence to be executed in its entirety.

In addition, a single functional block may be configured by a singlepiece of hardware, a single installation of software, or a combinationthereof. The functional configurations of the present embodiment arerealized by a processor executing arithmetic processing, and processorsthat can be used for the present embodiment include a unit configured bya single unit of a variety of single processing devices such as a singleprocessor, multi-processor, multi-core processor, etc., and a unit inwhich the variety of processing devices are combined with a processingcircuit such as ASIC (Application Specific integrated Circuit) or FPGA(Field-Programmable Gate Array).

In the case of having the series of processing executed by software, theprogram constituting this software is installed from a network orrecording medium to a computer or the like. The computer may be acomputer equipped with dedicated hardware. In addition, the computer maybe a computer capable of executing various functions, e.g., a generalpurpose personal computer, by installing various programs.

The storage medium containing such a program can not only be constitutedby the removable medium 100 of FIG. 4 distributed separately from thedevice main body for supplying the program to a user, but also can beconstituted by a storage medium or the like supplied to the user in astate incorporated in the device main body in advance. The removablemedium 100 is composed of, for example, a magnetic disk (including afloppy disk), an optical disk, a magnetic optical disk, or the like. Theoptical disk is composed of, for example, a CD-ROM (Compact Disk-ReadOnly Memory), a DVD (Digital Versatile Disk), Blu-ray (RegisteredTrademark) or the like. The magnetic optical disk is composed of an MD(Mini-Disk) or the like. The storage medium supplied to the user in astate incorporated in the device main body in advance is constituted by,for example, the ROM 12 of FIG. 4 in which the program is recorded or ahard disk included in the storage unit 19 of FIG. 4 or 5, etc.

It should be noted that, in the present specification, the stepsdefining the program recorded in the storage medium include not only theprocessing executed in a time series following this order, but alsoprocessing executed in parallel or individually, which is notnecessarily executed in a time series. Further, in the presentspecification, the terminology of the system means an entire apparatusincluding a plurality of apparatuses and a plurality of units.

The embodiments of the present invention described above are onlyillustrative, and are not to limit the technical scope of the presentinvention. The present invention can assume various other embodiments.Additionally, it is possible to make various modifications thereto suchas omissions or replacements within a scope not departing from thespirit of the present invention. These embodiments or modificationsthereof are within the scope and the spirit of the invention describedin the present specification, and within the scope of the inventionrecited in the claims and equivalents thereof.

INDUSTRIAL APPLICABILITY

The present invention is particularly useful for checking whether abeauty treatment is correctly given during the beauty treatment, and/orquantitatively evaluating the effect of the beauty treatment.

EXPLANATION OF REFERENCE NUMERALS

-   1 Measurement Device-   2 Network-   3 Server Group-   11 CPU-   12 ROM-   13 RAM-   14 Bus-   15 Input/Output. Interface-   16 Imaging Unit-   17 Input Unit-   18 Display-   19 Storage-   20 Communication Unit-   21 Drive-   30 Body-   31 Leg-   32 Hinge-   100 Removable Media-   111 Setting Processing Unit-   112 Acquisition Unit-   113 Detection Unit-   114 Correction Unit-   115 Comparison Unit-   116 Evaluation Unit-   117 Display Control Unit-   S Image Processing System

1-12. (canceled)
 13. An image processing device comprising: aninput/output interface; and at least one processor, wherein the at leastone processor executes operations including: detecting feature points ina facial image of an object included in an image taken from theinput/output interface; acquiring first distance information between thefeature points before a beauty treatment is performed; acquiring seconddistance information between the feature points at a second timing afterthe beauty treatment is performed; acquiring a difference value betweenthe first distance information and the second distance information; anddetermining whether the beauty treatment is correctly given or has beengiven based on a different value between the acquired distanceinformation.
 14. The image processing device according to claim 13,wherein the processor executes operations further comprising generatingan evaluation result in which the beauty treatment is expressed as ascore based on determined results, and wherein the input/outputinterface outputs the evaluation result.
 15. The image processing deviceaccording to claim 14, further comprising memory that stores a historyof the evaluation result, and wherein the input/output interface outputsthe history stored in the memory.
 16. The image processing deviceaccording to claim 14, wherein, when a number of consecutive days thebeauty treatment is evaluated to be at a certain level or above by theprocessor reaches or exceeds a preset target number of days, theinput/output interface outputs that the number of consecutive days hasreached or exceeded the target number of days.
 17. An image processingmethod executed by a computer, the method comprising: detecting featurepoints in a facial image of an object included in an image taken from aninput/output interface; acquiring first distance information between thefeature points before a beauty treatment is performed; acquiring seconddistance information between the feature points at a second timing afterthe beauty treatment is performed; acquiring a difference value betweenthe first distance information and the second distance information; anddetermining whether the beauty treatment is correctly given or has beengiven based on a different value between the acquired distanceinformation.
 18. The image processing method according to claim 17,further comprising generating an evaluation result in which the beautytreatment is expressed as a score based on determined results, andoutputting the evaluation result.
 19. The image processing methodaccording to claim 18, further comprising: storing a history of theevaluation result in memory, and outputting the history stored in thememory.
 20. The image processing method according to claim 18, furthercomprising outputting, when a number of consecutive days the beautytreatment is evaluated to be at a certain level or above by theprocessor reaches or exceeds a preset target number of days, that thenumber of consecutive days has reached or exceeded the target number ofdays.
 21. A non-transitory computer-readable storage medium storing aprogram that is executed by a computer, the program being executable tocause the computer to perform the following: detecting feature points ina facial image of an object included in an image taken from aninput/output interface; acquiring first distance information between thefeature points before a beauty treatment is performed; acquiring seconddistance information between the feature points at a second timing afterthe beauty treatment is performed; acquiring a difference value betweenthe first distance information and the second distance information; anddetermining whether the beauty treatment is correctly given or has beengiven based on a different value between the acquired distanceinformation.